Fault-tolerant linear solvers via selective reliability

interface. The Failable interface has methods for marking, unmarking, and checking whether the object’s data are allowed to experience bit flips. FTGMRES mark failability of the relevant objects on entry to the inner solver, and unmarks them on exit. We describe below how we implemented this high-level interface using the low-level application / OS interface presented in Section 9. Trilinos is built on the Petra framework of distributed linear algebra objects. Petra has two implementations: Epetra (Essential Petra), and Tpetra (Templated Petra). We use only Tpetra for our prototype, because Tpetra’s intranode parallel support library, Kokkos [6], has the necessary features to support our desired programming model. In particular, Kokkos allows us to intercept allocation and deallocation of large memory arrays, called compute buffers. Linear algebra objects such as sparse matrices, vectors, and preconditioners use compute buffers exclusively to store data on which they plan to execute parallel kernels. This lets us restrict where memory faults may occur, with minimal changes to the code of affected linear algebra objects. Kokkos also handles intranode parallelism in a generic way that encompasses both multicore CPU and GPU-based hardware. (In fact, this is why Kokkos needs control of memory allocation; it may need to place data on a GPU or other accelerator with a separate memory space from the CPU.) This lets our FT-GMRES prototype use hybrid parallelism (MPI and a threading library of our choice) without additional effort. Our software prototype currently works with multiple CPU-based threading libraries; we do not currently have GPU fault detection or injection capability, but this could be added at the level of the application / OS interface without changing our Trilinos modifications. We first extended the Kokkos interface to support marking or unmarking a compute buffer as “failable.” This operation directly invokes the application / OS interface discussed in Section 9. Our Kokkos extension gives us two ways to mark failability. We may either mark or unmark all subsequent allocations of compute buffers of a particular type (e.g., double) as failable, or mark or unmark a particular compute buffer. The first option lets us experiment with faults in Tpetra-based libraries without modifying their code. (For example, we can compute the sparse matrix A reliably, then intercept final assembly so that the matrix data are stored unreliably.) The second option – marking each buffer individually – lets us extend Tpetra linear algebra objects to implement the Failable interface. We then made Tpetra sparse matrices (CrsMatrix) and dense vectors (MultiVector) implement the Failable interface. Just like compute buffers, Failable objects may be marked or unmarked failable. Certain data in the object may experience memory faults only if the object is currently marked failable. Marking a Failable object consisting of compute buffers means marking some of its compute buffers. The object’s implementation gets to control which compute buffers may experience faults. For example, our sparse matrices only mark their entries, not the sparsity structure. We can also compose more complicated Failable objects out of simpler Failable objects. For example, an ILUT incomplete factorization preconditioner consists of two sparse matrices (the L and U factors); marking the preconditioner failable means marking the L and U factors